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New signal processing technologies have been developed to 
measure spatiotemporal neurocognitive processes of the human 
brain. In one experiment, application of these technologies 
produced measurements of distributed preparatory sets which 
predicted the accuracy of subsequent performance. In^ 
another experiment, neuroelectric changes were found in Air 
Force test pilots during the incipient stages of fatigue 
before behavior had severely degraded. 


THE METHOD OF EVENT-RELATED COVARIANCES (ERCs) 

Overview . 

We have been developing new methods for recording and 
analyzing task-related, spatiotemporal neurocognitive patterns 
from the unrelated electrical activity of the brain (refs. 1-14). 
Since neurocognitive processes are complex, we are concerned with 
spatiotemporal task-related activity recorded by many (currently 
up to 64) scalp electrodes in many (currently up to about 25) 
time intervals spanning a 4—6 second period extending from before 
a cue, through stimulus and response, to presentation of feedback 
about performance accuracy. Since goal-directed behaviors 
require integrated processing among many brain regions, we 
developed the method of event-related covariance (ERC) to measure 
salient aspects of the brain's distributed processing networks. 

The basis for ERC analysis lies with prior animal studies 
that have shown that when a brain region becomes involved in task 
performance, synchronization of a subset of neurons in that 
region is manifested as a change in the waveshape of its extra- 
cellularly recorded low frequency macropotentials (review in ref. 
8) . Since waveshape similarity and timing of macropotentials 
from different areas of the brain can be measured by covariance 
and correlation, these measures may characterize the spatial 
organization of coordinated functional activity of the areas 
involved in a goal-directed behavior. 
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Computing ERCs . 


A number of steps are currently performed in computing ERCs. 
The first pass reduces spatial smearing and then selects inter- 
vals and trials with task-related information to enhance the 
signal-to-noise ratio and reduce the amount of data prior to 
measuring ERCs. The second pass measures ERCs on band-pass- 
filtered, enhanced averages from the reduced data set. 

The steps include: 1) recording at least 50-100 trials of 

each task using at least 24 electrodes; 2) removing the effect of 
the reference channel and reducing spatial blur; 3) removing data 
with artifact contamination; 4) finding trials with consistent 
event-related signals and computing enhanced averages; 5) select- 
ing digital bandpass filters and intervals for measurement by 
examining ERPS, amplitude distribution maps and Wigner Distribu- 
tions; 6) computing multilag crosscovariance functions between 
all pairwise channel combinations of the enhanced averages in 
each selected analysis interval; 7) using the magnitude of the 
maximum crosscovariance function and its lag time as features 
characterizing the ERC; 8) estimating significance of ERCs by 
the standard deviation of the "noise" ERC; 9) graphing the most 
significant ERCs in each interval; and 10) statistically compar- 
ing ERC maps between conditions. 

The results of ERC analysis are expressed as color graphs. 
Since color photographs are not possible in these proceedings, 
the interested reader is referred to the published literature 
cited in this paper. 


Validation of ERCs . 

ERC analysis has been applied to data recorded from several 
experiments. The validity of the method is demonstrated in ana- 
lyses of visual stimulus processing and response execution inter- 
vals of a visuomotor task (refs. 5; 13). As predicted by neu- 
roanatomical theory and clinical neuropsychological studies, ERC 
patterns corresponding to the visual stimulus processing interval 
involved posterior sites that led anterior parietal sites and 
premotor sites (Fig. 1) . 

While ERC patterns appear to reflect the functional co- 
ordination of immediately underlying cortical areas, we must 
emphasize, however, that the actual neural sources of the ERC 
patterns are, in fact, not yet completely known. Determining the 
distributed source network producing the scalp ERC patterns is 
the major focus of our current technical efforts. 
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APPLICATION TO PREPARATION AND PREDICTING PERFORMANCE 


Procedure (refs. 5; 13) . 

Seven healthy, right-handed male adults participated in this 
study. A visual cue, slanted to the right or to the left, indi- 
cated to subjects to prepare to make a response pressure with the 
right or left index finger. One second later, the cue was fol- 
lowed by a visual numeric stimulus (number 1-9) indicating that a 
pressure of .1 to .9 kg should be made with the index finger of 
the hand indicated by the cue. Feedback indicating the exact 
response pressure produced was presented as a two-digit number 
one second after the peak of the response pressure. On a random 
20% of the trials, the stimulus number was slanted opposite to 
that of the cue, and subjects were to withhold their responses on 
these "catch trials". The next trial followed 1 sec after disap- 
pearance of the feedback. Subjects each performed several hundred 
trials, with rest breaks as needed. 

Twenty-six channels of EEG data, as well as vertical and 
horizontal eye-movements and flexor digitori muscle activity from 
both arms, were recorded. All single-trial EEG data were screened 
for eye-movement, muscle potential and other artifacts. Contam- 
inated data were discarded. 

Intervals used for ERC analysis were centered on major 
event-related potential (ERP) peaks. ERCS were computed between 
each of the 120 pairwise combinations of the 16 nonperipheral 
channels in intervals from 500 msec before cue to 500 msec after 
the feedback. 

Data sets were separated into trials in which subsequent 
performance was either accurate or inaccurate. Accurate and 
inaccurate performance trials were those in which the error 
(deviation from required finger pressure) was less than or 
greater than, respectively, the mean error over the recording 
session. 


Results and Discussion . 

ERC patterns during a 375-msec interval centered 687 msec 
post-cue (spanning the late Contingent Negative Variation; CNV) 
involved left prefrontal sites, regardless of subsequent accu- 
racy, as well as appropriately lateralized central and parietal 
sites (Fig. 2) . Inaccurate performance by the right hand was pre- 
ceded by a highly simplified pattern, while inaccurate perfor- 
mance by the left hand was preceded by a complex, spatially dif- 
fuse pattern. 

When the trials of each of the 7 subjects were classified by 
equations developed on the trials of the other 6 subjects, the 
overall discrimination was 59% (p<0.01) for right hand and 57% 
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(p<0.01) for left-hand performance. For the subject with the 
most trials, average classification of 68% (p<.001) for subse- 
quent right- and 62% (pc.Ol) for subsequent left-hand performance 
was achieved by testing a separate equation on each fifth of his 
trials, formed from the other four fifths. 

An ERC pattern involving covariances from midline parietal, 
left parietal, midline anterocentral and right frontal and 
anterocentral sites was common to feedback to both accurate and 
inaccurate right- and left-hand responses. When responses were 
inaccurate, however, the feedback pattern additionally included 
the midline and left-frontal sites. 

We suggest that our pre-stimulus ERC patterns characterize a 
distributed preparatory neural set related to the accuracy of 
subsequent task performance. This set appears to involve dis- 
tinctive cognitive (frontal), integrative-motor and lateralized 
somesthetic-motor components. The involvement of the left- 
frontal site is consistent with clinical findings that prepara- 
tory sets are synthesized and integrated in prefrontal cortical 
areas, and with experimental and clinical evidence indicating 
involvement of the left dorsolateral prefrontal cortex in delayed 
response tasks. A midline anterocentral integrative motor com- 
ponent is consistent with known involvement of premotor and sup- 
plementary motor areas in initiating motor responses. The finding 
of an appropriately lateralized central and parietal component is 
consistent with evidence from primates and humans for neuronal 
firing in motor and somatosensory cortices prior to motor 
responses. 

We further speculate that involvement of the midline antero- 
central site following feedback to both accurate and inaccurate 
performance may reflect "motor recalibration" consequent to feed- 
back information. Feedback-specific "updating" may be reflected 
by the involvement of the right prefrontal site for both accurate 
and inaccurate performance; behavioral verification, given feed- 
back about inaccurate performance, by the left prefrontal site. 


APPLICATION TO MEASURING EFFECTS OF INCIPIENT FATIGUE 


Procedure ( ref. 15). 

After learning and practicing a battery of tasks until their 
performance was stable on one day, each of five U.S. Air Force 
test pilots returned to the laboratory the next morning and per- 
formed the tasks for about 6 hours. Following a dinner break, 
they resumed task performance for an additional 6 to 8 hours. 

There were four tasks in the battery, including easy and 
difficult continuous and discrete visuomotor tracking tasks, a 
simple numeric memory task, and a difficult visuomotor memory 
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task (VMMT) . Since we expected that early neural signs of fatigue 
would be most evident during demanding tasks, we analyzed the 
VMMT first. This task required subjects to remember two continu- 
ously changing numbers, in the presence of numeric distractors, 
in order to produce precise finger pressures. Each trial con- 
sisted of a warning symbol followed by a single-digit visual 
stimulus to be remembered, followed by the subject's finger- 
pressure response to the stimulus number presented two trials 
ago, followed by a 2 -digit feedback number indicating the accu- 
racy of the response. For example, if the stimulus numbers in 
five successive trials were 8, 6, 1, 9, 4, the correct response 
would be a pressure of .8 kg when seeing the 1, . 6 kg for the 9, 
and .1 kg for the 4. To increase the task difficulty, subjects 
were required to withhold their response on a random 20% of the 
trials. These "no-response catch trials" were trials in which the 
current stimulus number was identical to the stimulus two trials 
ago. 


Trials early in the recording session with accurate finger 
pressures formed the "Alert" data set. Trials from early in the 
evening, when performance was just starting to decline, formed 
the "Incipient Fatigue" data set. For each subject, trials with 
relatively inaccurate responses were then deleted from the Inci- 
pient Fatigue data set so that the final Alert and Incipient 
Fatigue data sets consisted of trials with equivalently accurate 
performance. This crucial step allowed measurement of neuroelec- 
tric patterns associated with incipient fatigue while controlling 
for those due to variations in performance accuracy. 

EEGs were recorded with either 33 or 51 channels with a 
nylon mesh cap. Vertical and horizontal eye movements were also 
recorded, as were the responding flexor digitori muscle poten- 
tials, electrocardiogram and respiration. Three-axis Magnetic 
Resonance Image scans were made of 3 of the 5 subjects. 

Grand-average (over the five pilots) event-related poten- 
tials (ERPs) were time-locked to presentation of the numeric 
stimulus. Incipient-Fatigue ERPs were subtracted from Alert ERPs 
in order to highlight changes due to fatigue. Spatiotemporal 
neuroelectric patterns were then quantified by measuring ERCs 
between all 153 pairwise combinations of the 18 nonperipheral 
electrodes. ERCs were measured across brief segments of grand- 
average Alert-minus-Incipient-Fatigue subtraction ERPs. The 
first ERC interval was 500 msec wide and was centered 312 msec 
before the numeric stimulus. The next two ERC intervals were 187 
msec wide and were positioned with respect to the N125 and P380 
ERP peaks elicited by the numeric stimulus. 
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Results and Discussion. 


A number of significant Alert-minus-Incipient-Fatigue ERCs 
were found during the 500-msec prestimulus interval. Midline 
central, left parietal, left anteroparietal , right anterior 
parietal and right posterior parietal electrodes were the major 
ERC foci. There were no significant ERCs in the interval cen- 
tered at 62 msec post-stimulus. The ERCs computed over the P380 
no-response difference ERP were focused on the midline anterocen- 
tral, and right anterior and posterior parietal electrodes. 

Since ERCs are signs of functional interrelationships 
between brain areas, the ERC changes with Incipient Fatigue sug- 
gest that dynamic functional neural networks associated with 
specific cognitive functions are selectively affected during 
early fatigue. During the prestimulus interval, when subjects 
were maintaining the last two visually presented numbers in work- 
ing memory and preparing for the next stimulus, ERCs decreased in 
number in the Incipient Fatigue condition. The lack of ERC 
differences between Alert and Incipient Fatigue conditions during 
the interval centered a 62 msec suggests that the "exogenous" 
stages of visual stimulus processing are relatively unaffected by 
early fatigue. However, during the later post-stimulus interval 
of trials requiring an inhibition of the response, ERCs again 
decreased in number with Incipient Fatigue. ERCs involving 
anterocentral and right parietal electrodes characterized the 
difference between Alert and Incipient Fatigue conditions. Since 
precentral, central and parietal areas are implicated by neurop- 
sychological studies in the integration of numeric, visuospatial 
and visuomotor processes, the subtraction ERCs suggest a change 
in neural systems responsible for maintaining a representation of 
the magnitudes of the two visually presented numbers in working 
memory, and for inhibiting the response based on a comparison 
with working memory. 

Taken together, the data suggest that although neural sys- 
tems responsible for primary visual stimulus processing are rela- 
tively unaffected by incipient fatigue, cortical associative 
areas responsible for higher cognitive functions such as working 
memory rehearsal, preparation, and motor inhibition are altered 
prior to appreciable degradations in performance. 


CONCLUSIONS 

The bimanual task results demonstrate that the human brain, 
unlike a fixed-program computer, dynamically "tunes" its distri- 
buted, specialized subsystems in anticipation of the need to pro- 
cess certain types of information and take certain types of 
action. When these preparatory sets are incomplete or incorrect, 
subsequent performance is likely to be inaccurate. The fact that 
classification of performance accuracy improved when equations 
were formed and tested on the same subjects suggests that 
single-subject equations formed from large numbers of normative 
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trials may make ERC patterns useful for on-line prediction of 
subsequent behavior. 

The fatigue experiment results demonstrate the existence of 
"leading indicator" neuroelectric patterns which precede serious 
degradation of performance consequent to extended performance of 
a very difficult task. 

These studies demonstrate the potential of new neuroelectric 
signal processing technologies for measuring useful predictive 
information about the quality of performance. With further 
development, it should be possible to transition these technolo- 
gies from the pure research enviornment of the laboratory to 
application in flight simulators, and eventually in cockpits. 
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FINGER RESPONSE 

LEFT RIGHT 


Figure 1: View of the most significant, event-related covariance 
patterns from the wave at the peak of a finger response. The 
motor-related wave was measured during a 187-msec interval cen- 
tered on the peak of the left-hand and right-hand index finger 
pressures from theta-band filtered, seven-subject averages. The 
thickness of a line is proportional to tis significance (from .05 
to .00005). Line pattern indicates the time delay (lag time of 
maximum covariance) , and the arrow points from the leading to the 
lagging channel. ERC patterns for movement-registered timeseries 
also corresponded to prior functional neuroanatomical knowledge: 
the midline precentral electrode that overlies the premotor and 
supplementary motor cortices was the focus of all movement- 
related ERC patterns, and the other most significant ERCs 
involved pre- and post-central sites appropriately contralateral 
to the responding hand. Moreover, the pattern for the Motor 
Potential clearly reflected the sharply focused current sources 
and sinks spanning the hand areas of motor cortex. 
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COVARIANCE 
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Figure 2: View of the most significant (p<.05), between-channel 
CNV event-related covariance patterns from an interval 500 to 875 
msec after the cue for subsequently accurate and inaccurate 
left-hand (A) and right-hand (B) performance by seven right- 
handed subjects. The thickness of a line is proportional to its 
significance (from .05 to .005). Line pattern indicates whether 
covariance is positive (lighter lines) or negative (darker 
lines) . Covariances involving left-frontal and appropriately 
lateralized central and parietal electrode sites are prominent in 
patterns for subsequently accurate performance of both hands. 
Magnitude and number of covariances are greater preceding subse- 
quently inaccurate left-hand performance; fewer and weaker 
covariances characterize subsequently inaccurate right-hand per- 
formance. 
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